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(54) Representation of video images or 
scenes transmitted at reduced frame 
rate 

(57) Video image sequences or scenes 
with a relative movement between a 
pick-up sensor and the viewed scene, 
are transmitted at reduced frame rate. 
By interpolation and/or extrapolation 
on the basis of two transmitted frames 
intermediate frames are generated on 
the receiving end of the transmission 
line and inserted into the transmitted 
frame sequence. Owing to the 
movement, certain areas of the 
intermediate frame K are derived only 
from one reference frame O or the other 
reference frame N viz. hatched 
extrapolation areas. Other areas are 
derived from interpolation between the 
two reference frames viz. the 
cross-hatched area. Sudden variation of 
brightness levels can occur at the area 
boundaries. To avoid this, before 
reconstruction of the intermediate 
frames the mean brightness values for 
limited picture areas which represent 
the environment of at least one of the 
corresponding pels in two successive 
transmitted frames are determined. 
These brightness values are subtracted 
or divided and the brightness value in 
the extrapolation zone of the 
intermediate frames is corrected in 
accordance with the resulting 
differences or quotients. 
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SPECIFICATION 

Method for the representation of video images or scenes, in particular aerial images transmitted at reduced 
frame rate 

5 . 5 

The invention relates to a method for the representation of video image sequences or scenes with a relative 
movement between a pick-up sensor and the viewed scene, such scenes being transmitted at reduced frame 
rate. 

The invention refers to a method and an apparatus for the representation of video images, pictures or 
10 scenes, in particular aerial pictures with a relative movement between a pick-up sensor and the scene 10 
transmitted at reduced frame rate, in which by interpolation and/or extrapolation on the basis of two 
transmitted frames intermediate frames are generated and inserted into the transmitted frame sequence and 
in which the position of pels being identical in the two transmitted frames are determined from the relationship 
of the positions of at least two pels on different vertical and horizontal coordinates in each of the two 
15 transmitted frames as described in U.S. patent specification 4,496,972, to which reference is made. 15 
The intermediate frames are constructed of two reference frames recorded at different times (e.g. 1 second 
apart) whose frame information partly overlaps. 

In general with a frame movement in the x and y direction an intermediate frame K inserted in the sequence 
possesses three zones containing information of at least one of two reference frames. In the overlap zone of 
20 the two reference frames 0 and N information from both frame O and N isavailable.Two furtherzonesinclude 20 
information only present in one of the two reference frames O or N. In further zones there is no reference frame 
information. 

The intermediate frames are composed according to these conditions. Interpolation between thegrey 
values or frames 0 and N is possible in the overlap zone. By weighting the grey values depending on the time 
25 distance of the intermediate frame to be reconstructed from the reference frames a gradual changeover of the 25 
grey values from frame 0 to frame N can be reached. The zones for which information is available only from 
one of the two reference frames are taken directly from the relevant frame, i.e. without weighting depending 
on the time distance (extrapolation), compareGerman specification DE-OS34 08061 . As an alternative to the 
interpolation the extrapolation can be extended to the overlap zone. Zones with no reference frame 
30 information are given a predetermined grey value, e.g. medium grey value. 30 
If the lighting conditions between the recordings of two successive reference or transmitted frames change 
then the average brightness (i.e. the mean value of the grey value distribution) can differ in the extrapolation 
zone of the intermediate frames and in the interpolation zone or in parts of these zones. This difference affects 
the reproduction quality of the intermediate frames and has a negative visual effect particularly at junctions 
35 between thesezones, occurring asan abrupt change in the grey level. 35 
The object of the invention is to reduce the abrupt brightness changes at the junctions between interpolation 
zones and extrapolation zones where information from just one reference frame is available. 

The invention solves this task by a method and an apparatusfor the representation of video images, pictures 
or scenes, in particular aerial pictures with a relative movement between a pick-up sensor and the scene 
40 transmitted at reduced frame rate, in which by interpolation and/or extrapolation on the basis of two 40 
transmitted frames intermediate frames are generated and inserted into the transmitted frame sequence and 
in which the position of pels being identical in the two transmitted frames are determined from the relationship 
of the positions of at least two pels on different vertical and horizontal coordinates in each of thetwo 
transmitted frames, wherein before reconstruction of the intermediate frames for limited picture areas which 
45 represent the environment of the position of at least one of the corresponding picture elements (pels) in two 45 
transmitted frames the mean brightness values of these picture areas are determined, these brightness values 
are subtracted or divided and the brightness value in the extrapolation zone of the intermediate frames is 
corrected according to the resulting differences or quotients. 

Further features of the invention are described with reference to the drawings. 
50 The process involved expands the existing intermediate frame interpolation process so that the average 50 
brightness of the extrapolation and interpolation zones over the frame can be matched to reduce the 
brightness changes stated above and therefore avoids the negative visual effects. 

The present invention will be more readily appreciated by reference to the following detailed description 
when read in light of the accompanying drawing in which: 
55 Figure 7 shows the reconstruction principle of intermediate frames. 55 
Figure 2 shows an intermediate frame reconstructed from two referenced frames. 
Figure 3 is a block circuit diagram of intermediate frame reconstruction with adjusted brightness. 
Figure 1 shows the time relationship of two transmitted frames (O and N) and an intermediate frame (K) 
reconstructed on the receiving end of a transmission line and having the picture element (pel) positions A 0 ,A N 
50 and B 0 , B N respectively determined by measurement as well as the allocated picture elements C K , Co and C K , 60 
C N respectively determined by calculation based on a Unearmodel. 

The relationship between the pel position of an object and time, i.e. itsdisplacement between different 
frames, is determined by the laws of optical projection and geometry. It can be approximated by a linear 
characteristic, as shown in Figure 1. If the positions of corresponding pels in the two reference framesare 
65 known, for example positions A 0 , B 0 in frame O and A N , B N in frame N, the remaining pels of the picture field, 65 
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e.g. for the pels C 0 , C N can be determined from the relationship of the distance between the pels in accordance 
with the Ray Ratio Law. The relationship of the distance between the pel C 0 and C N with the coordinates x ' c0 , 
y'co and x' CN , y'cN and the relevant pel C K with the coordinates x' C k. y'ck in a frame K is characterized by a 
specific straight line between frame 0 and N and can be determined from the coordinate relationship, in a 
5 known manner as described in U.S. patent specification 4,496,972. 5 
Figure 2 shows the reference frames 0 and N which are displaced in time and according to their coordinates 
x and y. In addition the intermediate frame K to be reconstructed is shown by the broken lines. As illustrated, 
the overlapping zones indicated by square hatching contains information from both frames 0 and N. No 
overlap occurs in the areas with oblique hatchings. Information for these zones is available from one of the 
10 successive frames 0 and N, i.e. the zone with hatching slanting to the right is from frame N and the zone with 10 
the hatching slanting to the left is from frame O. Whilst the pels in the overlapping zone can be reconstructed 
by interpolation those in the non-overlapping zone are extrapolated using the information from one of the 
reference frames. Forzones for which no information is available predetermined grey values are given, for 
example a medium grey. 

15 The above shows that with major changes in the mean brightness values between successive reference 15 
frames a smooth changeover from the brightness of one frame to the next is only guaranteed in the 
overlapping zone where interpolation is possible. In the extrapolation zone only information from one of the 
reference frames is available. This leads to sudden changes in the grey value and therefore in the brightness at 
the junction between the interpolation zone and the adjacent extrapolation zones. Such abrupt grey changes 

20 

20 have a negative visual effect. 

Such abrupt changes are largely avoided by the process to be described below. This process uses the 
average brightness resulting from brightness averagings of limited picture areas which correspond in both 
reference frames. Displacement measurement techniques which can supply the positions of the matching pels 
in the frames or displacement values for those pels required for the interpolation of the intermediate frameand 

25 which operate according to the area correlation processes provide means brightness values of corresponding 25 
picture zones in both reference frames. 

The differences in the average brightness of corresponding picture areas are used to determinethe 
parameters of a function which is used to correct the brightness in the corresponding picture areas in the 
intermediate frame for any pel in the intermediate frame in the case of extrapolation. The parameters are 

30 determined by solving a system of linearequations containing the brightness differences as well as their 30 
coordinates in one of the reference frames. 

The process represents an improvement over the weighted interpolation of pels in intermediate framesin 
that exposure differences in successive reference frames no longer result in noticeable abrupt brightness 
changes at the junctions between interpolation and extrapolation zones. 

35 Details of the process are described in the following. 

From two successive reference frames 0 and N a number "n" of corresponding limited areas is selected. The 
actual number depends on the content of the reference frames. Corresponding limited pictu re areas are 
referred to hereinafter as picture area pairs. The size of picture areas is variable and can be e.g. 20 x 20 pels. 
Corresponding pairs of pels which are required to reconstruct the intermediate frames known from the 

40 displacement measurement techniques can be used as the centre of said picture area pairs. 

Such a picture area is shown in Figure 2 by the broken line around pel A. In the reference frame O the pel in 
the centre of this picture area has the coordinates A Q = (x oA , y oA ) and in reference frame N said pell has the 
coordinates A N = (x NA , y NA ). This figure shows further pels B, C and D to which corresponding pairs of picture 
areas in two successive frames are allocated. 

45 The determination of the correspondence between reference frame O and N resulting therefrom and the 45 
determination of the centre pels of the picture areas is carried out in block 1 2 of the circuit diagram shown in 
Figure 3. For each of the picture areas in the two successive frames O and N respectively a mean value of the 
brightness u oi and u Ni {Figure 3, block 14) is to be determined and thereafter the difference between these 
mean values (Figure 3, block 16) in accordance with the formula: 

50 

(D 

M-Di ~ M-Ni ~ M-oi 

The differences together with the corresponding frame coordinates of pels A Q , B 0 , C G , D Q ... or A N , B N , C N , D N ... 
are used in a system of preferably linear equations where the solution produces the parameters ai...a n forthe 
55 brightnesscorrection function (2) {block 18). 

In this, "n" represents the number of corresponding picture areas. 
Forexample (if n = 4): 

Ho (*o* y Q ) = a^oYo + a 2 x G + a 3 y 0 + a* w 

60 

Alternatively the coordinates in reference frame O (x 0 ,y 0 ) or in N (x N ,y N ) corresponding to the mean values can 
be used as reference variables in the formula, for example in (2). 

The grey value of a pel in an intermediate frame to be determined in the extrapolation zone can be calculated 
according to the extended interpolation formulae (3 and 4), block 20. 
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Ik< o) = l 0 <x 0 . Yo) + g (o, (K,N) m. d (x 0 ,Yo) (3) 

l K (NI = >n (x n . Yn) + g ,N, (K,r% D (x G , y Q ) (4) 

5 with for example 5 
IK_ 

g<°>(K,N) = N (5) 
K-N 

g (N) <K,N) = ~T\T^ (6) 
10 10 



The time-weighting factors g (o) and g {N> determine the time related continuous changeover in the brightness 
correction between reference frames O and N. The formulae {5 and 6} indicate the special case of time-linear 
changeover. 

The above describes the process on the basis of subtraction of the mean brightness values, A division of the 
15 mean brightness valuescan also be used according to the following formulae: 15 



Ui = fWH-oi ( 7 > 

Uj {x 0 , y Q ) = a!X o y 0 + a 2 x Q + a 3 y 0 + a* (8) 
20 20 
l K (o > = g (o ^(x 0 ,y 0 )l 0 (x 0 ,y 0 ) (9) 

lK {N, -g (N, ,(x 0 ,y 0 ) . 

In(x n ,Yn) HO) 
25 25 
Symbols 

u oi Mean brightness value in reference frame 0 relating to matching pel i. 

30 u Ni Mean brightness value in reference frame N relating to matching pel i. 30 

u Di Difference between corresponding mean values relating to matching pel i. 

v oi Quotientof corresponding meanvalues. 

35 35 
U o(x 0 /Yo) Function of brightness differences. 

i>D(XorYo) Function of brightness quotients. 

40 a 1 ,a 2 ,a 3 ,...a n Coefficients. 40 

x 0/ y 0 Pel coordinates forframeO. 

x N ,y N pell coordinates forframeN. 

45 45 
N Number of frames between two reference frames. 

K Position (0<K<N) of an intermediate frame. 

50 l K (0) Grey value of a pel in the intermediate frame in the position K when only a pel 50 

reference to the frame O exists (extrapolation zone). 

I K (N1 Grey value of a pel in the intermediate frame in the position K whereonly a pel 

reference exists to the reference frame N (extrapolation zone). 
55 55 
g (ol (K,N) Time-weighting factor according to the intermediate frame position K in 

respect to the reference frame 0. 

g (N1 (K,N) Time-weighting factor according to the intermediate frame position Kin 

60 respect to reference frame N. 60 
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1. A method and an apparatus for the representation of video images, pictures and scenes, in particular 
aerial pictures with a relative movement between a pick-up snesor and the scene transmitted at reduced frame 
rate, in which by interpolation and/or extrapolation on the basis of two transmitted frames intermediate 
frames are generated and inserted into the transmitted frame sequence and in which the position of pels being 

5 identical in the two transmitted frames are determined from the relationship of the positions of at least two 5 
pels on different vertical and horizontal coordinates in each of the two transmitted frames, wherein before 
reconstruction of the intermediate frames for limited picture areas which represent the environment of the 
position of at least one of the corresponding picture elements (pels) in two transmitted frames the mean 
brightness values of these picture areas are determined, these brightness values are subtracted or divided and 
10 the brightness value in the extrapolation zone of the intermediate frames is corrected according to the 10 
resulting differences or quotients. 

2. Method according to Claim 1, wherein the mean brightnesses of limited picture areas around the 
positions of at least four corresponding pels are used as the basis for extrapolation. 

3. Method according to Claim 1 or 2, wherein for the limited picture areas those pels are chosen which are 

1 5 selected to form the basis for reconstructing the intermediate frames. 1 5 

4. Method according to Claim 1 , wherein from at least two differences or quotients a function of the 
dependencies of the limited picture area is derived. 

5. Method according to Claim 4, wherein differences or quotients in the brightness values for further pel 
positions are calculated from said function with the coordinates of these further pel positions as variables. 

20 6. Method according to Claim 4, wherein the function is a polynom of the first order of the coordinates of 20 
one of the images transmitted. 

7. A method for the representation of video images in accordance with Claim 1 and substantially as 
described herein with reference to the accompanying drawings. 
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